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Overview 


CMMI 


Watts  Humphrey  asserted  as  early  as  1988: 

“/  can  walk  into  an  organization,  speak  to  a  few  members  of  a 
project,  and  know  within  10  minutes  the  maturity  level  of  the 
organization.  ” 

A  series  of  analyses  of  SEI  assessment  data  conducted  in 
1989-1990  by  Manuel  Lombardero  and  Alyson  Gabbard 
Wilson  supports  this. 

•  Derived  simple  binary  decision  trees  that  estimated  an 
organization’s  maturity  level  (ML  1-3)  with  low  rates  of  both 
false  positives  and  false  negatives 

•  CART  (Correlation  and  Regression  Tree  Analysis) 


Software  Engineering  Institute 


Carnegie  Mellon 


©2007  Carnegie  Mellon  University 


3 


Fundamental  Axioms  of  Statistical  Thinking 


CMMI 


All  product  development  and  services  are  a  series  of  interconnected 
processes. 

All  processes  are  variable. 

Understanding  variation  is  the  basis  for  management  by  fact  and 
systematic  improvement: 

•  understand  the  past-quantitatively 

•  control  the  present-quantitatively 

•  predict  the  future-quantitatively 
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Improvement  Must  be  for  the  Business 


CMMI 


Objectives 


Analysis  Indicators 

How  are  our  processes 
behaving? 


* 


Strategy  to 
accomplish 
the  objectives 
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Tasks  to  Accomplish 
objectives 


Task  1 
Task  2 
Task  3 

Task  n 


Success 

Criteria 


For 

Project  Manager 


Success  Indicators 

Have  our  objectives 
been  achieved? 


Roll-up  For 
Higher  Management 


w 


JDL 


Progress  Indicators  ->  Leading  Indicators 
How  is  the  “weather”  ahead? 
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What  Is  a  Process  in  Relation  to  Products  and  Cmml 
Services? 


Processes  defined  in  CMMI  are  “activities  that  can  be  recognized  as 
implementations  of  practices  in  a  CMMI  model.” 

They  may  also  be  thought  of  as  a  system  of  causes  that  includes  the 
people,  materials,  energy,  equipment,  and  procedures  necessary  to 
produce  a  product  or  service. 


Requirements 
&  Ideas 


I  - 

People  Material  Energy  Equipment  Procedures 

Q  Q  Q  Q  Q 


Work  activities 
Time 


Products  & 
Services 
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Distributions  Describe  Variation 


CMMI 


Populations  of  data  are  characterized  as  distributions  in  most  statistical 
procedures: 

•  expressed  as  an  “assumption”  for  the  procedure 

•  can  be  represented  using  an  equation 
Examples  of  distributions  you  may  come  across: 
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Central  Tendency  and  Dispersion 


CMMI 


Central  tendency  implies  location: 

•  middle  of  a  group  of  values 

•  balance  point 

•  examples  include  mean,  median,  and  mode 
Dispersion  implies  spread: 

•  distance  between  values 

•  how  much  the  values  tend  to  differ  from  one  another 

•  examples  include  range  and  (sample)  standard  deviation 

These  two  are  used  together  to  understand  the  baseline  of  a  process- 
performance  factor  and/or  outcome. 
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Sampling  the  Data 


Sampling  Considerations 

•  How  precise  do  we  need  the  answer  to  be? 

What  is  our  tolerable  margin  of  error? 

•  How  much  variation  do  we  expect  in  the  sample  data? 

What  is  the  sample’s  standard  deviation  or  proportion  percentage? 

•  How  confident  do  we  need  to  be  in  the  results? 

What  levels  of  “false  alarms”  and  “escapes”  are  we  willing  to  risk? 
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Measurement  Error  Threatens  Statistical 
Analysis 


CM  Ml 


How  big  is  the  measurement  error? 

What  are  the  sources  of  measurement  error? 

Is  the  measurement  system  stable  over  time? 

Is  the  measurement  system  capable? 

How  can  the  measurement  system  be  improved? 
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Impacts  of  Poor  Data  Quality 


CMMI 


Inability  to  conduct  hypothesis  tests  and  predictive  modeling 

Inability  to  manage  the  quality  and  performance  software  or  application 
development 

Ineffective  process  change  instead  of  process  improvement 

Improper  architecture  and  design  decisions  driving  up  the  lifecycle  cost 
and  reducing  the  useful  life  of  the  product 

Ineffective  and  inefficient  testing  causing  issues  with  time  to  market,  field 
quality,  and  development  costs 

Products  that  are  painful  and  costly  to  use  within  real-life  usage  profiles 
Bad  information  leads  to  bad  decisions! 
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High  Maturity  Practices  Require  Process 
Understanding  &  Statistical  Thinking 


Real  process  behavior  must  be  understood  before  making  conclusions 
about  the  performance  of  products  or  services. 

Ask  these  questions  to  find  out  about  real  process  behavior: 

•  What  is  the  normal  or  inherent  process  variation? 

•  What  differentiates  inherent  from  anomalous  variation? 

•  What  is  causing  the  anomalous  variation? 

Statistics  provides  the  methods  and  tools  needed  to  measure  and 
analyze  process  behavior,  draw  conclusions  (i.e.,  statistical  inferences), 
and  decide  next  steps. 
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Eleven  Frequently  Misinterpreted  ML  4-5 
Practices 


ML4-5  Practice 

%  of  Respondents 

Rank 

Cum  % 

OPP  1.5  -  Establish  PPMs 

49% 

1 

19% 

QPM  1.3  -  Select  Subprocesses  Statistically  Managed 

25% 

2 

28% 

QPM  1.2-  Compose  Project  Defined  Process 

23% 

3 

37% 

OPP  1.4  -  Establish  PPBs 

21% 

4 

45% 

OPP  1.1  -  Select  [Sub]  Processes 

19% 

5 

52% 

OID  1.2  -  Innovations 

19% 

5 

59% 

QPM  2.3  -  Monitor  [Subprocess]  Performance 

17% 

7 

66% 

QPM  1 .4  -  Manage  Project  Performance 

9% 

8 

70% 

QPM  2.2  -  Understand  Variation 

9% 

8 

73% 

OID  1.1  -  Improvement  Proposals 

9% 

8 

77% 

CAR  1.1  -  Select  Data  for  Analysis 

9% 

8 

80% 

Source:  Pat  O’Toole,  ATLAS  007.  53  provided  input  regarding  the  ML4-5  practices  that  most  lead  to  interpretation  issues. 
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OPP  Context  Diagram 
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OPP  SP  1.1  Select  Processes  -1 


CMMI 


Select  subprocesses  that  are  critical  to  achieving  the  organization’s 
objectives  and  to  predicting  whether  or  not  they  will  be  achieved. 

This  selection: 


•  is  driven  by  objectives  for  quality  and  process  performance  (OPP  SP  1 .3) 

•  influences  the  selection  of  measures  (OPP  SP  1 .2) 

•  is  based  on  an  analysis  of  PPBs  and  PPMs  and 
influences  their  coverage  (OPP  SP  1 .4-5) 


influences  which  subprocesses  projects  will  use  to  compose  their  defined 
process  and  statistically  manage  (QPM  SP  1 .2-3) 
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OPP  SP  1.1  Select  Processes  -2 


CMMI 


Selecting  appropriate  processes  must  be  based  on  clearly  defined 

•  lifecycle  models 

•  products  and  services 

•  organizational  business  objectives 

Selected  at  the  right  granularity 

•  large  projects:  lifecycle  phases  may  be  aggregates  of  subprocesses 

•  small  projects:  lifecycle  phases  may  be  the  subprocesses 

—  look  at  similar  activities  across  multiple  iterations,  builds,  and  projects 

Learn  what  works 

•  Use  data  to  determine  which  subprocesses  provide  insight  and  control  that  help 
projects  achieve  their  objectives 

—  “control  knobs” 

—  leading  indicators 
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OPP  SP  1.4  Establish  PPBs  -1 


CMMI 


Process-performance  baselines  are  built  from  project  data. 


Measures  of 
Process  Events 


Project  Estimating 
Planning,  and 
Management 


Organization's  Process- 
Performance  Baselines 
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OPP  SP  1.4  Establish  PPBs  -2 


ovum 


Process-performance  baselines  are 
derived  by  analyzing  the  collected 
measures  to  establish  a  distribution 
and  range  of  results  that  characterize 
the  expected  performance  for 
selected  processes  when  used  on  any 
individual  project  in  the  organization. 
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How  PPBs  are  Used 


CMMI 


Establish  and  verify  the  reasonableness  of  organizational  (OPP  SP  1.3) 
and  project  objectives  (QPM  SP  1.1) 

Compose  the  project’s  defined  process  (QPM  SP  1 .2) 

Establish  trial  limits  (QPM  SP  2.2) 

Identify  potential  sources  of  defects  and  problems  (CAR  SP  1 .2) 

Identify  opportunities  for  improvement  (OID  SP  1 .2) 

Evaluate  effects  of  a  change  on  process  performance  in  pilots  and  during 
or  after  deployment  (e.g.,  a  before-and-after  comparison)  (CAR  SP  2.2; 
OID  SP  1 .3,  2. 2-2. 3) 
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PPB  Lessons  Learned 


CMMI 


PPBs: 

•  are  based  on  data  available  at  a  frequent  enough  rate  and  timely  fashion 

•  are  based  on  measures  that  have  common  operational  definitions  across 
projects  to  support  organizational  consolidation 

•  for  time  ordered  as  well  as  non-time  ordered  data 

•  address  subprocess  effort,  cycle  time,  quality,  and  cost 

•  can  include  data  from  non-stabilized  subprocesses  (but  flagged  as  such) 

Some  organizations  try  to  start  with  only  one  set  of  PPBs  (and  PPMs)  but 
later  conclude  that  they  need  different  sets  for  different  product  lines. 

Projects  may  establish  their  own  PPBs. 

•  provides  for  improved  estimating  and  prediction  within  project 
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OPP  SP  1.5 


PPMs 


CMMI 


PPMs  are  used  to  estimate  or  predict  the  value  of  a  process-performance 
measure  from  the  values  of  other  process  and  product  measurements. 

PPMs  typically  use  process  and  product  measurements  collected 
throughout  the  lifecycle  to  estimate  progress  toward  achieving  objectives 
which  cannot  be  measured  until  later  in  the  lifecycle. 

These  models  are  defined  to  provide  insight  and  to  provide  the  ability  to 
predict  critical  process  and  product  characteristics  that  are  relevant  to 
business  value. 

The  result  of  using  a  PPM  to  make  a  prediction  often  takes  the  form  of  a 

prediction  interval  (as  opposed  to  a  single  point). 
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How  PPMs  are  Used 


CMMI 


Establishing/verifying  reasonableness  of  organization  (OPP  SP  1.3)  and 
project  objectives  (QPM  SP  1.1-3) 

Determining  whether  project  is  on  track  to  meeting  its  objectives  (QPM  SP 
1.4) 

Analyzing/predicting  impact,  benefit,  and  ROI  when  evaluating/selecting: 

•  Defects  and  problems  for  analysis  (CAR  SP  1.1) 

•  Action  proposals  for  implementation  (CAR  SP  2.1) 

•  Process  improvement  proposals  for  implementation  (OID  SP  1.1, 
1.4) 

•  Candidate  innovations  (OID  SP  1 .2,  1 .4) 

Evaluating  effects  of  a  change  on  process  performance  to  see  if  predicted 
performance  is  met  (CAR  SP  2.2,  OID  SP  2. 2-2. 3) 
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Essential  Ingredients  of  PPMs  -1 


CMMI 


Relate  the  behavior  or  circumstance  of  a  subprocess  to  an  outcome. 

Predict  future  outcomes  based  on  possible  or  actual  changes  to  factors 
(e.g.  support  “what-if  analysis). 

Use  factors  from  one  or  more  subprocesses  to  conduct  the  prediction. 

•  The  factors  used  are  preferably  controllable  so  that  projects  may  take  action 
to  influence  outcomes. 

Are  statistical  or  probabilistic  in  nature  rather  than  deterministic  (e.g.  they 
account  for  variation  in  a  similar  way  that  QPM  statistically  accounts  for 
variation;  they  model  uncertainty  in  the  factors  and  predict  the  uncertainty 
or  range  of  values  in  the  outcome). 
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Essential  Ingredients  of  PPMs  -2 


CMMI 


High  maturity  organizations  generally  possess  a  collection  of  process- 
performance  models  that  go  beyond  predicting  cost  and  schedule 
variance,  based  on  Earned  Value  measures,  to  include  other  performance 
outcomes. 

Specifically,  the  models  predict  quality  and  performance  outcomes  from 
factors  related  to  one  or  more  subprocesses  involved  in  the  development, 
maintenance,  service,  or  acquisition  processes  performed  within  the 
projects. 

Process-performance  models  must  provide  useful  insight  for  projects  to 
use  them  as  value-added  tools. 
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Example  Subprocesses  to  Be  Modeled 


CMMI 


Lifecycle  phase  subprocesses 

•  e.g.  Req’ts,  Architecture,  Design,  Code  and  Test  (cycle  time,  quality 
performance  or  defect  density,  productivity,  staff  attributes,  or  risk  indices) 

Those  contributing  to  resolution  of  inquiries  or  actions  related  to  key 
communication  interface  subprocesses 

•  e.g.  with  suppliers,  customers,  partners 
Inspection  and  peer  review  subprocesses 

•  e.g.,  preparation  rates,  review  rates,  defects  found  densities 

Those  contributing  to  downtime  of  essential  parts  of  the  project  environment 

•  e.g.,  computing  resources,  test  equipment,  specialized  tools  and  compilers 
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Types  of  Models 


CMMI 


Basic  statistical  prediction  models 

•  basic  statistics  to  predict  outcomes 

Monte  Carlo  simulation  and  Optimization  models 

•  automated  “what-if”  analysis  of  uncertain  factors  and  decisions  in  a  spreadsheet 

Process  simulation  models 

•  actually  model  process  activities  w/computer 

System  dynamics  models 

•  same  as  above  but  with  real-time  feedback  loops 

Probabilistic  models 

•  prediction  using  laws  of  probability  instead  of  statistics 

Reliability  growth  models 

•  fitting  test  failure  data  to  known  distributions  for  enabling  predictions  of  future 
failure  experience 
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Basic  Statistical  Prediction  Models 


CMMI 
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Chi-Square 
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ANOVA: 


•  What  are  the  escaped  defect  densities  (e.g.,  defects  per  KSLOC)  for  each 
type  of  peer  review  (e.g.  inspection,  walkthrough),  and  are  the  densities 
statistically  different  by  peer  review  type? 
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Monte  Carlo  Simulation 


CM  Ml 


Allows  modeling  of  variables  that  are  uncertain  (e.g.  put  in  a  range  of 
values  instead  of  single  value) 

Enables  more  accurate  sensitivity  analysis 

Analyzes  simultaneous  effects  of  many  different  uncertain  variables  (e.g. 
more  realistic) 

Eases  audience  buy-in  and  acceptance  of  modeling  because  their  values 
for  the  uncertain  variables  are  included  in  the  analysis 

Establish  confidence  levels  to  outcomes  (e.g.  supports  risk  management) 
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Example:  Building  a  Business  Case  from  the  Cmml 
Defect  Model 


Phase  Found 

Phase  lnjecte< 

Pre- 

Existing 

(SEG) 

\ 

Existing 

(Contr) 

\oposal 

>| 

Plann^ra 

Design 

Impl^^fnt 

formal 

Test 

Release 

Total 

Cost  to 
Correct  Prior 
to  Delivery 

Cost  to  Correct 
After  Delivery 
(based  on 
Industry 
Standards) 

Cost  Avoidance 

Proposal 

#  of  Defect 

1 

177 

^  1764 

^  0 

^  0 

0 

0 

1942 

Rework  (Hrs) 

4 

32.8 

467.65 

0 

0 

0 

0 

0 

504.45 

S45,400.50 

S4, 540, 050.00 

S4, 494, 649. 50 

Planning 

#  of  Defect 

0 

1 

14 

1019 

0 

0 

0 

0 

1034 

Rework  (Hrs) 

0 

1 

3.4 

172.07 

0 

0 

0 

0 

176.47 

SI  5.882.30 

SI. 588. 230.00 

SI, 572, 347. 70 

Design 

#  of  Defect 

0 

26 

8 

14 

2187 

0 

0 

0 

2235 

Rework  (Hrs) 

0 

17.35 

320.91 

7.75 

851.42 

0 

0 

0 

1197.43 

S107.768.70 

S10, 776, 870.00 

SI  0,669, 101 .30 

Implement 

#  of  Defect 

1 

28 

15 

27 

61 

4460 

0 

0 

4592 

Rework  (Hrs) 

0.1 

30 

5.85 

0.71 

244.5 

2521 .99 

0 

0 

2803.15 

S252.283.50 

S2, 522, 835.00 

S2, 270, 551. 50 

Formal  Test 

#  of  Defect 

0 

11 

0 

2 

17 

285 

608 

0 

923 

Rework  (Hrs) 

0 

42 

0 

A  0.12 

229.42 

139.1 

559.65 

0 

970.29 

S87, 326.10 

S873, 261.00 

S785, 934.90 

Release 

#  of  Defect 

0 

0 

59| 

™  0 

4 

0 

961 

1028 

Rework  (Hrs) 

0 

0 

9  6# 

0 

.  43 

^^^8.3 

0 

152.24 

213.15 

S19.183.50 

8191,835.00 

SI  72,651 .50 

Total 

#  of  Defect 

2 

243 

imo 

1062 

Jkes 

608 

961 

11754 

Rework  (Hrs) 

4.1 

123.15 

00 

* 

ro 

180.65 

13G&.34 

2669. 39 

^559.65 

152.24 

5864.94 

S527, 844.60 

S20, 493, 081.00 

519,965,236.40 
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Process  Modeling  Simulation 


CMMI 


Grade  1 


Grade  2 


Grade  3 


Station  1 


Station  2 


Shipping 


pBach  activity,  such  as 

“Staging”,  is  assigned 
information  about  capacity 
NoH  time  to  perform,  and 
information  about  input 
Vaueue  lengths. 


Operation  Criteria 

1  Grade  1  items  cannot  be  mixed  with  Grades  2  and/or  3  darifig  staging  or  at  processing  stations 

2  Grades  2  and  3  may  be  mixed  during  staging  and  at  processing  stations 
3.  Grade  1  items  can  only  be  processed  by  Stations  1  and  2 


Adapted  from  ProcessModel,  Inc. 
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Bayesian  Belief  Network  Quality  Model 


CMMI 


\ 


Defects  in  Product 

0.48  _ 

0.4 

0.32 

0.24  _ 

0.16 

0  .08  _ 

0.0 

-r(T  1  TTTrm — 

It 

ooooo  H^1 

0)CT)0)CT)Cn-»-^-J 

o 

Testing  Quality 


0.32 

0.24 

0.1G 

0.08 


Prob  of  Finding  Defect  in  Test 


Z 


Defects  Found  During  Test 


/We  can  predict  the 
'  probability  of  finding  a 
defect  during  a  test,  by 
learning  what  the  quality  of 
testing  is. 

Also,  we  can  predict 
defects  found  by  learning 
more  about  the  expected 
incoming  defect  level  and 
our  ability  to  find  defects 
.with  testing! 


From  AgenaRisk,  Ltd 
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Six  Sigma/Modern  Statistics  Essential  to  OPP 


CMMI 


SP1.1  Processes 

SP1.2  Measures 

SP1.3  Objectives 

SP1.4  Baselines 

SP1.5  Models 

[=: 

Big  Y  Business  Goal-to-Vital  x  Process; 

Processes  driving  central  tendency  and  variation 

Critical  Parameter  Management;  CTQ  factors;  Root 
Cause  Analysis  of  subprocess  factors 


KJ  Analysis®;  Analytic  Hierarchy  Process;  Categorical 
Survey  Data  Analysis;  Six  Sigma  Scorecards 

Control  Charts;  Graphical  Summaries  in  Minitab;  Central 
Tendency  and  Variation;  Confidence  and  Prediction 
Intervals 

ANOVA;  Regression;  Chi-Square;  Logistic  Regression; 
Monte  Carlo  Simulation;  Discrete  Event  Process 
Simulation;  Design  of  Experiments;  Response  Surface 
Methodology;  Multiple  Y  Optimization;  Probabilistic 
Models 
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QPM  Context  Diagram 


CMMI 


Quantitatively  Manage  the  Project 


OPP 


Predictions  of 
Quality  and 
Process 
Performance 


Organization’s 

Measurement 

Repository 


Subprocess 

Capability 


CAR,  OID 
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Relevant  Terminology 


CMMI 


Statistical  management  (QPM  SG  2) 

Management  involving  statistical  thinking  and  the  correct  use  of  a  variety 
of  statistical  techniques,  such  as  run  charts,  control  charts,  and  prediction 
intervals. 


Quantitative  management  (QPM  SG  1) 

The  process  of  using  data  from  statistical  and  other  techniques  to  manage 
the  project 

•  predict  whether  it  will  be  able  to  achieve  its  quality  and  process- 
performance  objectives 

•  identify  what  corrective  action  (if  any)  should  be  taken 
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QPM  SP  1.2  Compose  the  Defined  Process 


CMMI 


With  what  process  composition  can  the  project  best  meet  its  objectives? 

•  selects  the  subprocesses  that  helps  it  best  achieves  its  goals 

•  may  try  different  compositions  of  subprocesses 

-  build  a  PPM  of  each  candidate  composed  process  to  predict  if  the  goals 
will  be  achieved 

—  if  no  candidate  process  is  predicted  to  achieve  the  project’s  objectives 

o  CAR  can  help  find  improvements  to  the  process  to  achieve  the 
objectives. 


o  OID  can  help  identify  innovations  that  will  enable  meeting  the 
objectives. 
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One  Approach  to  PDP  Composition 


CMMI 


Problem 

Produce  a  product  in  a  given  period  of  time  with  an  acceptable  minimum  level  of 
quality. 

Step  1 

Examine  the  baselined  subprocesses: 

•  Are  there  appropriate  lifecycle  processes? 

•  Are  there  options  with  differing  performance  characteristics? 

Step  2 

Load  a  Monte  Carlo  simulation  with  candidate  subprocesses,  their  data  (PPBs), 
and  constraints  (e.g.,  subprocess  A  won’t  work  with  subprocess  B). 

Step  3 

Establish  optimization  priorities.  Run  the  Monte  Carlo  simulation. 

Step  4 

Evaluate  the  simulation  outputs  to  determine  if  the  candidate  subprocesses  will 
solve  the  problem. 
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Monte  Carlo  Optimization  for  Quality 


CMMI 


Performance  Graph 


"BEST  SOLUTION 


Frontier 


Values  of  Variables: 

RD  Decision:  1 
RR  Decision:  4 
Design  Decision:  2 
DR  Decision:  4 
Code  Decision:  4 
CR  Decision:  2 
UT  Decision:  1 
IT  Decision:  3 
ST  Decision:  3 
AT  Decision:  1 

Objective:  Overall  Goal:  Mean:  8 
Requirement  Feasible 
Requirement:  TEST01 
Requirement:  TEST02 
Requirement:  TEST03 
Additional  details  may  be  found  below... 


1200000-1- 


1040000-  - 


Objective 


880000 


720000 


560000 


400000 


/  Requirement  Feasible 


/  Requirement  Infeasible 


ReScalei 


100  200  300  400 

Simulation 


500 


600 


700 


This  solution  of  process  composition  is 
optimized  with  first  priority  of  quality  and 
secondary  priority  on  cycle  time.  Run 
additional  simulations  reflecting 
alternative  optimization  priorities. 
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Summary  of  Monte  Carlo  Results 


CMMI 


Subprocesses 

Optimize  for 

Cycle  Time 

Quality 

Requirements  Development 

Traditional 

Traditional 

Requirements  Review 

Email  Routing 

Sampling  Inspections 

Design 

SA/SD 

OOD 

Design  Review 

Email  Routing 

Sampling  Inspections 

Code 

Code  Generation  w/Reuse 

Code  Generation  w/Reuse 

Code  Review 

Email  Routing 

Walkthrough 

Unit  Test 

Ad  Hoc 

Ad  Hoc 

Integration  Test 

Hybrid 

Hybrid 

System  Test 

Production  Hardware 

Production  Hardware 

Acceptance  Test 

Low  Intensity 

Low  Intensity 

Results  (95%  Confidence  results  won’t  exceed) 

Cycle  Time 

171 

185 

Quality  Rework  Costs 

$487,000 

$354,000 

Overall  Costs 

$7,935,000 

$841 ,000 
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QPM  SP  1.3  Select  the  Subprocesses  that  Will  Cmmi 
Be  Statistically  Managed  -1 


The  project  selects  the  subprocesses  it  will  statistically  manage  to  help  it 
meet  its  objectives  (QPM  SP  1.1). 

•  Decision  is  based,  in  part,  on  the  organization’s  selected 
subprocesses  for  organizational  process-performance  analyses 
(OPP  SP  1.1). 

—  And  on  the  subprocesses  composing  the  PDP  (QPM  SP  1 .2). 

•  The  selected  subprocesses  then  become  the  “subject”  of  the  SPs  of 
SG  2  Statistically  manage  subprocess  performance 
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QPM  SP  1.3  Select  the  Subprocesses  that  Will  Cmmi 
Be  Statistically  Managed  -2 


Determine  important  subprocess 
attributes 

•  Which  attributes  provide  insight 
into  subprocess  performance? 

•  Which  attributes  most  affect 
downstream  performance? 

•  Perform  sensitivity  analyses. 

•  Improve  PPM  coverage  of 
upstream  subprocesses  (at 
project  and  organizational  level) 
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SP  1.4  Manage  Project  Performance 


CMMI 


Periodically  evaluate  progress  toward  achieving  the  project’s  quality  and 
process-performance  objectives  by  doing  the  following: 

•  review  the  capability  of  each  selected  subprocess 

•  review  the  actual  results  achieved  against  the  interim  performance  targets 
for  that  phase 

•  review  results  of  suppliers  against  their  performance  targets 

•  calibrate  PPMs  to  estimate  progress 

If  not  on  track,  evaluate  options  for  corrective  action  by  using  adjusted 
PPBs  and  PPMs  to  predict  the  effects  an  option  will  have  (“what  if?”); 
thereby  determining  which  is  the  best  option  for  getting  back  on  track. 
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QPM  SP  2.2  Apply  Statistical  Methods  to 
Understand  Variation 


CMMI 


Establish  and  maintain  an  understanding  of  the  variation  of  the  selected 
subprocesses  using  the  selected  measures  and  analytic  techniques. 

1 .  Establish  trial  natural  bounds  for  subprocesses  having  suitable  historical 
performance  data. 

2.  Collect  data,  as  defined  by  the  selected  measures,  on  the  subprocesses  as  they 
execute. 

3.  Calculate  the  natural  bounds  of  process  performance  for  each  measured  attribute. 

4.  Identify  special  causes  of  variation. 

5.  Analyze  the  special  cause  of  process  variation  to  determine  the  reasons  the 
anomaly  occurred. 

6.  Determine  what  corrective  action  should  be  taken  when  special  causes  of  variation 
are  identified. 

7.  Recalculate  the  natural  bounds  for  each  measured  attribute  of  the  selected 
subprocesses  as  necessary. 
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Control  Charts 


CMMI 


Specification 
Limits  \ 


mean  +  3a 


Upper  Control  Limit  (UCL) 


mean  -  3a 


-  Mean 

Lower  Control  Limit  (LCL) 


- > 

Event  Time  or  Sequence 


Control  Limits  - *-  Set  by  analyzing  historical  process  data 

(Voice  of  the  process) 

Specification  Limits  - ^  Set  by  customer,  engineer,  etc. 

(Voice  of  the  customer) 
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Understanding  Variation  and  Level  4  -1 


ovum 


A  common  misconception  is  that  because  you  are  using  control  charts, 
you  are  ML4.  Using  control  charts  is  only  one  statistical  method  that  aids  in 
understanding  process  variation. 

Instead,  ML4  (more  precisely,  QPM)  requires  that  you  understand  process 
variation  for  selected  subprocesses  and  quantitatively  manage  the  project 
based  on  that  understanding. 

The  understanding  of  process  variation  becomes  the  basis  for  the 
following: 

•  determining  when  to  take  corrective  action 

•  predicting  future  performance 

•  determining  whether  the  subprocess  is  capable  of  achieving  its  quality  and 
process  performance  objectives 
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Understanding  Variation  and  Level  4  -2 


CMMI 


When  control  limits  are  too  wide,  sources  of  variation  are  easily  masked  (i.e., 
the  process  is  not  in  control  though  it  may  appear  to  be)  and  future 
performance  cannot  be  predicted. 

To  improve  predictability,  investigate  sources  of  variation  by  examining  the 
following: 

•  upstream  subprocesses 

•  various  subgroups  of  the  data  (e.g.,  data  grouped  by  source  or  work  product 
size) 

•  the  measures  being  used 

This  investigation  may  lead  to  changes  in  which  subprocesses  and  attributes 
are  targeted  for  control. 

Wherever  possible,  teams  should  use  their  own  data  for  the  same  subprocess 
in  order  to  better  understand  and  stabilize  a  subprocess’s  performance. 
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Revising  Control  Limits  UnMasks  Other 
Sources  of  Variation 


CMMI 


Individuals  (X)  Chart 
Inspection  Design  Review-Hours 


Individuals  (X)  Chart 
Inspections  Design  Review-Hours 


Inspection  Sequence 
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Notes  on  Using  Control  Charts 


CMMI 


Some  organizations  want  staff  to  check  for  normality  before  using  XmR  or 
XbarR 


•  Wheeler:  don’t  need  to  worry  about  it 

•  Stoddard:  easy  to  do,  why  not? 

If  data  isn’t  normal  should  we  transform  it  before  placing  on  a  control 
chart? 


•  Stoddard:  usually  don’t  gain  much  from  doing  this,  and  we  easily  lose 
intuition  of  what  the  control  limits  mean 

Common  pitfalls 

•  arbitrarily  setting  control  limits 

•  freezing  control  limits 

•  removing  points  outside  limits  to  show  stability  (but  without  investigating 
these) 
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QPM  SP  2.3  -  Monitor  Performance  of  the 
Selected  Subprocesses 


CMMI 


Monitor  the  performance  of  the  selected  subprocesses  to  determine  their 
capability  to  satisfy  their  quality  and  process-performance  objectives,  and 
identify  corrective  action  as  necessary. 


1 .  Compare  the  quality  and  process-performance  objectives  to  the  natural 
bounds  of  the  measured  attribute. 

2.  Monitor  changes  in  quality  and  process-performance  objectives  and 
selected  subprocess’  process  capability. 

3.  Identify  and  document  subprocess  capability  deficiencies. 

4.  Determine  and  document  actions  needed  to  address  subprocess 
capability  deficiencies. 
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Stability,  Capability,  and  Voices 


CMMI 


SP  2.2  is  about  determining  the  voice  of  the  subprocess  and  achieving  a 
stable  subprocess. 

•  A  stable  subprocess  is  the  state  in  which  all  special  causes  of  process 
variation  have  been  removed  and  prevented  from  recurring  so  that  only  the 
common  causes  of  process  variation  remain. 

SP  2.3  is  about  comparing  the  voice  of  the  subprocess  to  the  voice  of  the 
customer  to  determine  if  the  subprocess  is  capable. 

•  A  capable  subprocess  is  a  process  that  can  satisfy  its  specified  product 
quality,  service  quality,  and  process-performance  objectives. 

•  Thus,  having  a  stable  subprocess  is  prerequisite  to  SP  2.3. 
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A  Capable  Process? 


CMMI 


C=i  Voice  of  the  Process  i=> 


c 

ZD 


o 

O 


o 
c 
< D 


3 

C7 

CD 


LCL=  36.08  CL=  45.06  UCL=54.04 


Product  Service  Staff-Hours 

<  'Voice  of  the  Customeri=> 
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Stability,  Capability,  Limits,  and  Models  -1  cmml 

The  Basics 


Start  with  a 
stabilized  process 

or  at  least  one  for  which  we 
know  the  central  tendency 
and  dispersion 


Which  determines  a 

Use  a  PPM  to  prediction  interval  for 

..  .  that  attribute 

predict  a 
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Stability,  Capability,  Limits,  and  Models  -2 

What  can  happen 


CMMI 


The  predictor 
process  can  vary 
(mean  or  range) 


PPM  produces 
a  new 
prediction 


Which  suggests 
we’re  no  longer  on 
track  to  meeting  our 
objectives 


^  Prediction 
■  Interval 

A— I 

I _ 
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Stability,  Capability,  Limits,  and  Models  -3  cmml 

The  Tolerance  Interval 


Tolerance  interval  shows 


how  much  the  predictor 
process  can  vary  while 
keeping  the  prediction 
interval  within  the 
specification  limits 


—  Software  Engineering  Institute  Carnegie  Mellon 
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T 
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Six  Sigma/Modern  Statistics  Essential  to  QPM  Cmmi 


SGI  Quantitatively  I  \ 
Manage  the  Project 


KJ  Analysis;  Analytic  Hierarchy  Process;  Categorical 
Survey  Data  Analysis;  Six  Sigma  Scorecards;  Big  Y 
Business  Goal-to-Vital  x  Process;  Process  Mapping 
Methods  and  Value-Stream  Analysis; 

Processes  driving  central  tendency  and  variation; 
Critical  Parameter  Management;  CTQ  factors;  Root 
Cause  Analysis  of  Sub-process  factors;  Cockpit 


SG2  Statistically  | - >  Control  Charts;  Graphical  Summaries  in  Minitab;  Central 

Manage  Subprocess  *  Tendency  and  Variation;  Confidence  and  Prediction 
Performance  Intervals;  ANOVA;  Regression;  Chi-Square;  Logistic 

Regression;  Monte  Carlo  Simulation;  Discrete  Event 
Process  Simulation;  Design  of  Experiments;  Response 
Surface  Methodology;  Multiple  Y  Optimization; 
Probabilistic  Models 
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CAR  Application  at  Higher  Levels 


CMMI 


During  project  execution,  issues  arise  that  CAR  can  help  solve: 

•  No  process  can  be  composed  that  will  meet  objectives  (QPM  SP  1 .2) 

•  Project  will  not  achieve  its  objectives  (QPM  SP  1 .4) 


Subprocess  control  limits  (or  prediction  interval)  spread  too  far  apart  to  be 
of  much  value  in  control  and  prediction  (QPM  SP  2.2) 

Special  causes  of  variation  (QPM  SP  2.2) 

Subprocess  not  capable  (QPM  SP  2.3) 
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CAR  SP  1.1  Select  Defect  Data  for  Analysis 


CMMI 


When  selecting  (sets  of)  defects  or  problems  for  further  analysis,  consider  the 
following: 

•  impact 

•  frequency  of  occurrence 

•  similarity  between  defects 

•  cost  of  analysis 

•  time  and  resources  needed 

•  safety  considerations 

•  ROI 

PPBs  and  PPMs  can  be  useful  for:  (1)  identifying  defects  or  problems  to  work  on, 
(2)  analyze  root  causes,  (3)  predicting  the  impact  and  ROI  of  potential  solutions, 
and  (4)  confirming  the  impact  after  deployment. 

Your  cost/benefit  analyses  should  also  consider  impacts  to  the  capability  of  the 
process. 
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Six  Sigma/Modern  Statistics  Essential  to  CAR  Cmmi 


SP1.1  Select  Defect 
Data  for  Analysis 


SP1.2  Analyze 
Causes 


SP2.1  Implement 
the  Action 
Proposals 


o 

■=> 

■=> 


Measure  Phase  (within  DMAIC  or  DMAD(O)V)  tools  and 
methods;  Models  provide  insight  to  the  areas  of  defect 
data  to  concentrate  on 

Root  Cause  Methods,  e.g.  Ishikawa  Diagrams,  statistical 
hypothesis  tests  to  determine  if  segments  are  different 


Piloting;  Comparative  Studies;  Technological  and  Cultural 
Change  Management  techniques 


SP2.2  Evaluate  the 
Effect  of  Changes 


SP2.3  Record  Data 


>  Before  and  After  studies  and  Hypothesis  tests;  Survey 
categorical  data  analysis;  compare  to  results  of  prediction 
models 

>  Study  results;  Lessons  Learned  shared  across  the 
organization;  Institutional  learning 
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OID  Context  Diagram 


CMMI 
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How  OID  SP  1.1  and  1.2  Interrelate 


CMMI 


Process  improvement 
proposals  submitted  from 
engineering,  management, 
suppliers,  customers,  etc. 


Collect  and  analyze  process 
improvement  proposals  to 
determine  costs,  benefits, 
risks,  and  barriers  (OID  SP 
1.1). 

•  Use  PPMs  to  predict 
the  performance  of  the 
revised  process  and  to 
facilitate  cost/benefit 
analysis. 


Identify  areas  where  improvements 
would  be  beneficial  and  the  technologies 
and  innovations  that  target  those  areas 
(OID  SP  1.2): 

•  Use  PPBs  and  PPMs  to  identify 
potential  areas  for  improvement. 

•  Identify  innovations  to  address  the 
areas  needing  improvement. 

•  Use  PPMs  to  predict  impacts, 
costs,  and  benefits  of  particular 
innovations. 

•  Create  process  improvement 
proposals  for  promising 
innovations. 
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OID  SP  1.1  Collect  and  Analyze  Improvement  Cmml 
Proposals  -1 

Sources  for  improvement  proposals  include: 

•  Causal  analysis  activities 

•  Process-  and  technology-improvement  proposals 

•  Investigation  of  innovative  improvements 

•  Analysis  of  PPBs  and  PPMs 

•  findings  and  recommendations  from  process  appraisals 

•  analysis  of  customer  and  end-user  problem  and  satisfaction  data 

•  analysis  of  data  about  project  performance  compared  to  objectives 

•  examples  of  improvements  that  were  successfully  adopted  outside  the  organization 

•  analysis  of  technical  performance  measures 

•  results  of  process  and  product  benchmarking  efforts 

•  feedback  on  previously  submitted  proposals 

•  spontaneous  ideas  from  managers  and  staff 

Mature  organizations,  because  of  their  refined  segmentation  of  process  and 
technology,  can  work  in  a  more  focused  fashion  to  gather  their  benchmarking  data. 
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OID  SP  1.1  Collect  and  Analyze  Improvement 
Proposals  -2 


CMMI 


Improvement  proposal  costs  and  benefits  are  evaluated  using  criteria: 

•  contribution  toward  meeting  the  organization’s  quality  and  process- 
performance  objectives 

•  effect  on  mitigating  identified  project  and  organizational  risks 

•  effect  on  related  processes  and  associated  assets 

•  cost  of  defining  and  collecting  data  that  supports  the  measurement  and 
analysis  of  the  process-  and  technology-improvement  proposal 

•  expected  life  span  of  the  proposal 

PPMs  can  be  used  to  predict  what  performance  would  result  from  process 
changes,  thus,  facilitating  cost  benefit  analyses. 

•  We  will  show  an  example  of  this  for  SP  1 .2. 
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OID  SP  1.2  Identify  and  Analyze  Innovations 


CMMI 


Identify  and  analyze  innovative  improvements  that  could  increase  the 
organization’s  quality  and  process  performance. 

1 .  Analyze  the  organization’s  set  of  standard  processes  to  determine  areas 
where  innovative  improvements  would  be  most  helpful. 

2.  Investigate  innovative  improvements  that  may  improve  the  organization’s 
set  of  standard  processes. 

3.  Analyze  potential  innovative  improvements  to  understand  their  effects  on 
process  elements  and  predict  their  influence  on  the  process. 

4.  Analyze  the  costs  and  benefits  of  potential  innovative  improvements. 

5.  Create  process-  and  technology-improvement  proposals  for  those 
innovative  improvements  that  would  result  in  improving  the  organization’s 
processes  or  technologies. 

6.  Select  the  innovative  improvements  to  be  piloted  before  broadscale 
deployment. 

7.  Document  the  results  of  the  evaluations  of  innovative  improvements. 
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The  Role  of  Analyses  in  OID  SP  1.2 


CMMI 


The  subpractices  of  SP  1.2  mention  performing  various  analyses. 

These  analyses  are  performed  to  determine  which  subprocesses  are 
critical  to  achieving  the  organization's  quality  and  process-performance 

objectives  -  either  as  direct  contributors  and/or  as  leading  indicators. 

•  Use  PPBs  and  PPMs  to  determine  which  factors  to  target  for  innovation 
(i.e.,  to  determine  which  subprocesses  to  improve). 

•  When  a  factor  is  selected  as  a  target  for  innovation,  use  PPBs  and  PPMs  to 
evaluate  the  impacts,  costs,  and  benefits  of  candidate  innovations  (“what 
ifs”)  within  that  area. 

The  PPBs  and  PPMs  mentioned  above  may  already  exist,  or  they  may 
need  to  be  developed  to  support  performing  these  analyses. 
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Investigate  Innovative  Improvements 


CMMI 


Investigating  innovative  improvements  involves  the  following: 

•  systematically  maintaining  awareness  of  leading  relevant  technical  work  and 
technology  trends 

•  periodically  searching  for  commercially  available  innovative  improvements 

•  collecting  proposals  for  innovative  improvements  from  projects  and  the 
organization 

•  systematically  reviewing  processes  and  technologies  used  externally  and 
comparing  them  to  those  used  within  the  organization 

•  identifying  areas  where  innovative  improvements  have  been  used 
successfully,  and  reviewing  relevant  data  and  documentation 

—  Difficult-to-meet  objectives  may  have  led  some  projects  to  compose  a 
defined  process  (QPM  SP  1.2)  having  promising  performance 
characteristics 

•  identifying  improvements  that  integrate  new  technology  into  products  and 
project  work  environments 
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Using  PPBs  and  PPMs  to  Select  an  Area  to  Improve  -1 


CMMI 


First,  some  terminology: 

Defect  density  at  release  (DD)  is  the  number  of  defects  per  unit  of  product 
size  (e.g.,  KLOC)  in  the  product  at  release. 

Requirements  volatility  (RV)  is  the  rate  at  which  requirements  change  once 
design  begins,  e.g.,  %  requirements  statements  (or  use  cases)  changed, 
annualized  per  year. 

Design  complexity  (DC)  is  the  number  of  unique  subtrees  in  the  calling- 
tree  hierarchy*. 

Effectiveness  of  quality  checks  (QC)  is  a  measure  (e.g.,  a  count)  of  the 
items  that  have  been  addressed  on  a  check  sheet  for  requirements  and 
design  verification. 

Staff  turnover  (ST)  is  a  measure  of  project  team  churn,  e.g.,  #  of  staff 
replaced  on  the  project  team,  annualized  per  year. 


*  See  “Design  Complexity  Measurement  and  Testing,”  by  Thomas  J.  McCabe  and  Charles  W. 
Butler,  CACM,  Dec.  1989,  for  more  information. 
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Using  PPBs  and  PPMs  to  Select  an  Area  to  Improve  -2 


ovum 


Assume  an  organizational  objective  for  product  quality: 

•  Defect  density  at  release  (DD)  <  0.4  defects/size  unit 
is  not  often  met. 

Use  a  formal  evaluation  process  (DAR)  to  determine  which  factors  (i.e., 
“areas”)  to  focus  on. 

Step  1.  Establish  criteria  for  evaluating  factors  that  impact  DD. 

•  Example  criteria  include: 

—  contribution  to  (i.e.,  influence  on)  DD 

—  potential  for  innovations  in  that  factor 

—  potential  costs  and  risks  (or  opportunities)  associated  with  changing 
that  factor 
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Using  PPBs  and  PPMs  to  Select  an  Area  to  Improve  -3  CMMI 


Step  2.  Identify  factors  that  impact  DD. 

Perform  a  regression  analysis  creating  a  PPM  that  shows  the  relationship 
between  DD  and  several  contributing  factors: 

•  requirements  volatility  (RV) 

•  design  complexity  (DC) 

•  effectiveness  of  quality  checks  (QC) 

Of  course,  such  a  PPM  might  already  exist. 

Re-run  the  regression  analysis,  as  appropriate,  to  evaluate  additional 
promising  factors,  e.g.,  staff  turnover  (ST). 
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Using  PPBs  and  PPMs  to  Select  an  Area  to  Improve  -4CJWJVfl 


Results  for:  Defect  Density  Process  Performance  Model 
Regression  Analysis:  Defect  Density  versus  RV,  DC,  QC, 


The  regression  equation  is 
Defect  Density  =  389  +  2.12  RV  +  5.32  DC 


-  24.1  QC 


Predictor 

Coef 

SE  Coef 

T 

P 

Constant 

389.17 

66.09 

5.89 

0.000 

RV 

2.125 

1.214 

1.75 

0.092 

DC 

5.3185 

0.9629 

5.52 

0.000 

QC 

-24.132  . 

w  1.869 

-12.92 

0.000 

Evaluate  whether 
requirements  volatility 
(RV),  design  complexity 
(DC),  and  effectiveness 
of  quality  checks  (QC) 
have  an  impact  on  DD. 

(An  analysis  with  staff 
turnover  is  not  shown 
here.) 


PPMs  help  select  areas 
to  target  for  innovation. 

Analysis  of  Variance 


R-Sq(adj)  =  85.9% 


Note  the  p-values 
for  RV  vs.  DC,  QC. 


Source 

DF 

SS 

MS 

Regression 

3 

12833.9 

4278.0 

Residual  Error 

25 

1848.1 

73.9 

Total 

28 

14681.9 

F 

57.87 


0.000 
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Using  PPBs  and  PPMs  to  Select  an  Area  to  Improve  -5CMMi 


Step  3.  Evaluate  the  factors  using  the  established  criteria. 

How  do  we  evaluate  “contribution  to  DD?”  Several  analysis  tools  are 
available  to  help.  For  example,  Crystal  Ball  (www.crvstalball.com) 
provides: 

•  Tornado  Chart  -  screens  out  factors  with  little  contribution  potential  to  allow 
for  a  more  vigorous  simulation-based  sensitivity  analysis. 

•  Sensitivity  analysis  -  performs  a  Monte  Carlo  simulation  to  derive  estimates 
of  the  correlations  between  factors  and  the  outcome  (more  broadly, 
“assumptions”  and  a  “forecast”). 

—  Results  in  a  bar  chart  that  ranks  factors  according  to  their  influence  on 
the  variability  of  an  outcome  (DD  in  this  case) 

—  A  bar’s  direction  indicates  whether  the  correlation  is  positive  or 
negative.  (See  next  slide.) 
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Using  PPBs  and  PPMs  to  Select  an  Area  to  Improve  -6CJVfJWI 


Here  is  the  result  of  a  Crystal  Ball-based  sensitivity  analysis.  Note  which 
factors  have  the  largest  impact  on  DD. 


Design  Complexity  (DC) 
Req’ts  Volatility  (RV) 
Quality  Check  (QC) 

Staff  Turnover  (ST) 


0.0% 


21  0% 


42.0% 


63.0% 


65.2% 

-7.2^ 

2.1 )% 


25.5% 


From  the  above  steps  (with  possible  iteration),  the  conclusion  is  design 
complexity  (DC)  is  the  most  promising  factor  to  target  for  improvement. 


Detailed  design  is  thus  targeted  for  innovations  that  will  help  reduce  DC. 
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Using  PPBs  and  PPMs  to  Evaluate  Candidate 
Innovations  -1 


A  formal  evaluation  (DAR)  is  typically  performed  on  candidate  innovations. 

As  part  of  that  evaluation,  PPMs  play  an  important  role  in  predicting 
impacts,  costs,  and  benefits  from  deploying  the  candidate  innovation. 

This  helps  determine  whether  piloting  the  innovation  (and  its  possible 
subsequent  deployment)  is  worth  pursuing. 

The  role  that  a  PPM  might  play  is  illustrated  with  an  example. 

Step  1.  Estimate  a  90%  confidence  interval  for  DD  for  a  process  employing 
the  current  detailed  design  subprocess  (pre-innovation). 

•  Use  a  Monte  Carlo  simulation  with  the  PPM  generated  earlier  to  evaluate 
what  to  expect  for  DD  under  the  current  conditions. 
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Using  PPBs  and  PPMs  to  Evaluate  Candidate 
Innovations  -2 


CMMI 


Input  ranges  and  most  likely  value  are  specified  for  each  factor  in  the  PPM 
(reflecting  the  PPBs  for  these  factors). 


Defect  Density 

= 

389 

+ 

2.12 

RV 

+ 

5.32 

DC 

■ 

24.1 

QC 

0.327124 

20% 

20 

7 

15% 

15 

2 

Min 

20% 

20 

7 

Most  Likely 

45% 

30 

8 

Max 

These  inputs  drive  the  simulation,  resulting  in  a  confidence  interval  for  DD 
for  the  current  detailed  design  subprocess. 
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Using  PPBs  and  PPMs  to  Evaluate  Candidate 
Innovations  -3 


CMMI 


1.000.000  Trials 


Frequency  View 


997.542  Displayed 


Defect  Density  per  KSLOC 


This  translates  into 
90%  confidence  that 
defect  density  will  not 
exceed  0.42  in  the 
current  process. 
However,  that  is  not 
good  enough  to  make 
customers  happy! 


1,000 


0.28  0.29  0.30  0.31  0.32  0.33  0.34  0.35  0.36  0.37  0.38  0.39  0.40  0.41  0.42  0.43  0.44  0.45  0.46  0.47 
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Using  PPBs  and  PPMs  to  Evaluate  Candidate 
Innovations  -4 


ovum 


Step  2.  Estimate  a  90%  confidence  interval  for  DD  for  the  new  detailed 
design  subprocess  (simulation  of  the  process  assuming  the  innovation 
was  made). 

•  Use  a  Monte  Carlo  simulation  with  the  PPM  generated  earlier,  but  with  a 
modified  range  for  DC  reflecting  the  behavior  expected  from  the  new 
detailed  design  subprocess. 

Step  3.  Repeat  for  other  candidate  innovations,  if  any. 

Step  4.  Select  an  innovation  to  further  pilot. 


Note:  A  similar  analysis,  perhaps  performed  in  a  more  straightforward  way, 
could  also  be  performed  in  SP  1.1,  especially  for,  but  not  limited  to,  non¬ 
trivial  incremental  improvements. 
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Using  PPBs  and  PPMs  to  Evaluate  Candidate 
Innovations  -5 


ovum 


Input  ranges  and  most  likely  value  are  specified  for  each  factor  in  the  PPM 
(reflecting  the  PPBs  for  RV  and  QC,  but  with  a  modified  estimate  for  DC). 


Defect  Density 

= 

389 

+ 

2.12 

RV 

+ 

5.32 

DC 

m 

24.1 

QC 

0.327124 

20% 

20 

7 

15% 

7 

2 

Min 

20% 

10 

7 

Most  Likely 

45% 

20 

8 

Max 

The  candidate  innovation  should 
reduce  design  complexity  from  a  range 
of  15-30  to  this  new  range  of  7-20 

These  inputs  drive  the  simulation,  resulting  in  a  confidence  interval  for  DD 
for  the  current  detailed  design  subprocess. 
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Using  PPBs  and  PPMs  to  Evaluate  Candidate 
Innovations  -6 


CMMI 


1.000.000  Trials 


Frequency  View 


997.479  Displayed 


Defect  Density  per  KSLOC 


This  translates  into 
90%  confidence  that 
defect  density  will  not 
exceed  0.37  in  the 
improved  process. 

This  represents  an 
improvement  over  the 
previous  defect  density 
of  0.42  and  should 
make  customers 
happier! 


1,000 


0.24  0.25  0.26  0.27  0.28  0.29  0.30  0.31  0.32  0.33  0.34  0.35  0.36  0.37  0.38  0.39  0.40  0.41  0.42 
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Six  Sigma/Modern  Statistics  Essential  to  OID 


CMMI 


SGI  Select 
Improvements 


>Six  Sigma  Big  Y  to  Vital  x  semi-annual  workshops; 
Business  Goal  simulation  and  optimization  models; 
Benchmarking;  Capability  data  sharing;  Theory  of 
Inventing  (TRIZ)  methods;  Usage  of  performance  models 
to  identify  the  major  opportunities  for  improvement  with 
innovation;  Assumption  Busters;  Empowered  innovative 
thinking;  Incentives  for  Innovation;  Strong  Teaming  for 
Innovation;  Various  decision  models  such  as  AHP,  Pugh 
Method,  Probabilistic  decision  trees 


SG2  Deploy 
Improvements 


Process  and  Design  FMEA;  Organizational  Readiness 

>for  Change;  Change  Agents;  Sponsors;  Champions; 
Influence  Leaders;  Adoption  Curve;  Piloting;  Risk-based 
deployment;  Before  and  After  comparisons  with 
Hypothesis  tests;  Results  compared  to  prediction 
models;  Proactive  mitigation  of  risks 
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High  Maturity  Is  Management  with  a 
Navigation  System 


CMMI 


Measurement  is  used  routinely  and 
proactively: 

•  Are  we  confident  we  know 
where  we  are,  where  we  are 
going,  and  our  performance 
outcomes  (quantitative 
understanding)? 

•  Do  we  understand  variation? 

Use  measurement  results  to  answer 
the  questions  “Will  we  be 
successful?,”  “Are  our  customer 
expectations  and  what  we  are 
capable  of  doing  aligned?,”  and  “What 
if  we  were  to  do  something  different?” 
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CMMI  is  a  Set  of  Interrelated  Practices 


CMMI 


Interrelationships  are  key  to  understanding  levels  4  and  5. 

•  These  interrelationships  are  not  always  obvious. 

•  By  understanding  these  interrelationships,  the  richness  of  levels  4  and  5 
becomes  evident. 


•  The  interrelationships  become  evident  in  the  informative  material  -  read  it! 


You  cannot  abandon  your  lower  level  practices  as  you  become  more 
mature — you  need  to  evolve  them  and  incorporate  them. 
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Current  SEI  Courses  Related  to  High  Maturity  Cmmi 


Course 

Description 

Understanding  CMMI  High 
Maturity  Practices 

CMMI  level  4-5  concepts,  practices,  and 
implementation  (various  statistical  methods  are 
introduced) 

Appraising  CMMI  High 
Maturity  Organizations 
(proposed) 

Making  judgments  in  support  of  an  appraisal  of  an 
organization’s  implementation  of  CMMI  levels  4-5 

Measuring  for 
Performance-Driven 
Improvement  1 

Statistical  methods  and  tools  involved  in  analyzing 
data  for  product  and  process  design  and 
optimization  based  on  the  DMAIC  roadmap 

Measuring  for 
Performance-Driven 
Improvement  II 

Statistical  methods  and  tools  involved  in  analyzing 
data  for  product  and  process  design  and 
optimization  based  on  the  DFSS  roadmap 
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Conclusion 


CMMI 


Understanding  variation  is  the  basis  for  management  by  fact  and 
systematic  improvement. 

High  maturity  organizations  use  appropriate  statistical  and  other 
quantitative  methods  to 

•  understand  past  quality  and  process  performance 

•  predict  future  quality  and  process  performance 

•  target  areas  for  improvement  and  evaluate  the  impact  of  proposed 
improvements 

•  focus  on  innovation  and  how  to  be  more  competitive 

The  SEI  is  upgrading  its  training  curriculum  to  help  organizations  further 
develop  their  knowledge  and  skills  related  to  implementing  and  appraising 
CMMI  high  maturity  practices. 
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